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ABSTRACT 


The Monterey Security Architecture (MYSEA) provides trusted security services, allowing 
users to access information at different sensitivity levels at the same time. The MYSEA server 
enforces a mandatory access control policy to ensure that users can only access data for which 
they are authorized. We would like to know the consequences of the MYSEA design on the per¬ 
formance of the MYSEA system. In particular, have the MYSEA trusted processes introduced 
any design bottlenecks into the system? 

The objective of this thesis is to analyze the performance of selected aspects of MYSEA 
and, when applicable, identify system performance bottlenecks. In the absence of bottlenecks, 
our secure system performance study can be interpreted as characterizing the “cost of security” 
in a multilevel security context. We analyze the overhead associated with MYSEA by targeting 
and benchmarking its components and services. We deployed the netperf tool as a MYSEA 
service, to observe costs associated with IPSec, the MYSEA trusted proxy and communication 
among servers in the MYSEA Federation. Our benchmark tests provided useful insights to the 
performance overhead introduced by MYSEA’s design and highlighted the cost of security of 
selected aspects in MYSEA. 
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CHAPTER 1: 

Introduction 


Demand exists for the mandatory enforcement of confidentiality and integrity policies, espe¬ 
cially in the military and business domains. The need to share information across different 
sensitivity domains necessitates the development of multilevel distributed security architectures 
that enforce information flow policies, while maintaining usability and efficiency. Military and 
commercial systems are currently unable to provide high assurance support for multilevel secu¬ 
rity policy enforcement 0. With adversaries and hackers performing increasingly sophisticated 
attacks, the lack of robust security measures are a cause for concern due to the risk of leakage 
of sensitive information, corruption of critical data and denial of service. New trusted com¬ 
puting approaches involving interoperable system security features and standardized security 
mechanisms are required to secure mission-critical systems. 

MYSEA provides simultaneous access to different security classifications of information using 
standardized commercial-off-the-shelf components. Mandatory enforcement of confidentiality 
and integrity policies necessitates the design of distributed architectures to control information 
flow between systems while providing for security controls to protect against malicious code 
and attacks from adversaries. 

1.1 Motivation 

The confidentiality, integrity and availability (CIA) of data, applications and networks are vital 
to any organization. Mandatory access controls, where data is segregated into security classifi¬ 
cations and is only accessible to suitably authorized personnel are a way of addressing security 
concerns associated with malicious software. The need to protect sensitive data and allow se¬ 
lective sharing of information between users necessitates the development of high assurance 
trusted systems. For users to embrace the security controls imposed, systems need to be ef¬ 
ficient and user-friendly. Therefore, there is a need to investigate the system performance of 
MYSEA, to identify and analyze the potential bottlenecks. 

1.2 Purpose of Study 

Various trusted security components are part of MYSEA. These amend the underlying platform 
to provide authorized users with simultaneous access to information of different classification 
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levels, introducing necessary overhead to the system. There is a need to investigate the “cost of 
security” and study the performance overhead associated with high assurance systems following 
this design, compared with systems operating traditional servers. This is especially crucial for 
multi-user distributed systems where scaling and user-friendliness become important factors in 
user-acceptance and ergonomic security, both of which are stated MYSEA goals. 

To create a trusted security environment, the MYSEA design introduces some (perhaps sig¬ 
nificant) overhead that may impact performance. Every TCP connection (for User Datagram 
Protocol (UDP), every packet) from the multilevel Local Area Network (LAN) is processed, 
verified and cross-referenced against access control permissions by a trusted subject. This in¬ 
spection takes place in different processes in the server architecture. Key Performance Indica¬ 
tor (KPI)s associated with the system need to be measured and their impacts assessed. The “cost 
of security” in terms of efficiency, performance and responsiveness is an important factor in the 
application and risk analysis of security controls. Thus, measurement of the KPIs is needed 
before an informed tradeoff between the performance penalties associated with implementing 
MYSEA and the security requirements and policies mandated. 

The goal of our study is to analyze the performance of selected aspects of MYSEA and, when 
applicable, identify system performance bottlenecks. System usability is determined by many 
factors, one of which is system latency measured by the end-to-end time taken for a system 
to react to a user’s command. Another important factor is to gauge the throughput of the sys¬ 
tem, measuring the bytes per second processed by the system. Benchmarking the distributed 
MYSEA system will help to reveal potential design inefficiencies so that, in the long run, steps 
can be taken to improve its performance and efficiency. A bottleneck occurs when the perfor¬ 
mance of a system is limited by a number of components or resources, thus reducing the overall 
throughput or increasing the latency experienced by a user. In the absence of bottlenecks, our 
secure system performance study can be interpreted as characterizing the “cost of security” in 
a multilevel security context. 

1.3 Thesis Organization 

The remainder of this thesis is organized as follows: Chapter [2] provides the background on 
MYSEA and an overview of its trusted components and services, which in combination form 
the multilevel secure distributed system. It also provides a detailed discussion of the design of 
MYSEA and the design considerations that impact its performance. Chapter [3] outlines the ob¬ 
jectives of the thesis by identifying the high-level and detailed requirements for benchmarking 
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MYSEA, including the criteria for selection of a benchmark tool. These requirements provide 
the basis for the selection of the KPIs that were measured. This chapter also analyses the various 
benchmark tools for consideration; compares the pros and cons of each tool and selects a tool 
to benchmark MYSEA. It describes the benchmark methodology used to ensure that tests were 
conducted fairly, accurately and without bias. Chapter [4] describes the test environment, its test 
configurations and the test plan used for the execution of the benchmarks. Chapter [5] deals with 
the actual implementation of the benchmarks and analyzes the results collected by the various 
benchmark tests. Chapter [6] concludes by summarizing the results, discussing related work and 
providing recommendations for future work. 
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CHAPTER 2: 

Monterey Security Architecture Overview 


MYSEA, The Monterey Security Architecture, is a distributed client-server architecture com¬ 
posed of existing Government-off-the-shelf (GOTS) and Commercial-off-the-shelf (COTS) soft¬ 
ware, open-source components, and (relatively few) special-purpose, high-assurance compo¬ 
nents Q@@0- MYSEA provides a trusted computing operating environment for enforcing 
information-flow policies in a multilevel LAN. MYSEA allows vertical integration of applica¬ 
tion security requirements with underlying security services; an existing Quality of Security 
Service model 0 and framework are applied to the integrated security structure. In addition, 
MYSEA supports a secure and unforgeable bidirectional connection, called a trusted path, be¬ 
tween the user and the trusted elements of the systems of the system, as well as trusted channels 
between devices. 


MYSEA enforces confidentiality and integrity policies formalized in the Bell and La-Padula 
(BLP) and Biba security policy models respectively. The high assurance MYSEA Servers en¬ 
force the security policies and control the information that a user can access at a particular sensi¬ 
tivity level. These servers host various open-source or commercial application servers (i.e., web 
server and mail server), providing services along with multilevel views of information to clients. 
MYSEA currently supports Simple Mail Transfer Protocol (SMTP) 0, Internet Message Ac¬ 
cess Protocol (IMAP) (7J, Voice over Internet Protocol (VoIP) |8j, Web Distributed Authoring 
and Versioning (WebDAV) 0, Hypertext Transfer Protocol (HTTP) [ 10] and Wiki [11]. 


MYSEA clients are commodity PCs equipped with specialized Trusted Path Extension (TPE) 
devices. TPEs are inserted between unsecured client workstations and the MYSEA server. 
Every network connection from a MYSEA client to the Multilevel Security (MLS) servers is 
labeled, thus permitting the enforcement of the security policies for data entering MYSEA 
Server. Together, the MYSEA Servers and TPEs form the Trusted Computing Base (TCB). 


Section |2T] provides the MYSEA design overview and Section |2.2| describes a typical usage 
scenario of how clients gain access to the services hosted on MYSEA server; Section |23] pro¬ 
vides a detailed description of the important components and services in MYSEA and possible 
sources of performance overhead. Section [2~4| outlines the security policy enforced in MYSEA 
and related design decisions. Section [23] describes the current MYSEA prototype. 
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2.1 MYSEA Design Overview 

MYSEA is designed to provide application services in a high assurance, distributed environ¬ 
ment, while protecting users from malicious code and other attacks. The design of MYSEA 
allows mandatory policy enforcement mechanisms to be allocated to a few specialized high- 
assurance trusted components, while encompassing other low assurance commercial compo¬ 
nents and applications to support operational functionality. 

The key design principles of MYSEA are (1) security dependencies must be partially ordered; 
(2) the ordering of trust and trustworthiness must be present and (3) the principle of least priv¬ 
ilege must be enforced. A federation of highly trustworthy MLS servers provides the locus of 
policy enforcement, mediating all accesses to resources in the system. The MYSEA servers 
host various open-source or commercial services such as mail and web services. The MYSEA 
federation is designed to be a robust and scalable foundation supporting expansion: the scala¬ 
bility requirement for MYSEA is to support up to 100 clients with concurrent user sessions at 
100 different security levels. 

To establish a secure communication channel between clients and MYSEA servers, trusted 
TPEs authenticate and disambiguate clients. All network traffic flows in and out of MYSEA 
are tracked to determine the security level of the client making the request. This results in an 
implicit labeling of all network traffic. 

MYSEA servers and TPEs collectively enforce the mandatory security policies, forming a 
Trusted Computing Base (TCB). The MYSEA TCB is designed to facilitate evaluation by 
TCB subsets p~2| . TCB subsets allow for the partition of a complex system into smaller dis¬ 
joint trusted subsets, each of which resides in its own protection domain and enforces a security 
policy subset upon the subjects and objects under its control [^2j. The composition of each 
TCB subset is small and less complex, and can be evaluated individually for policy enforce¬ 
ment correctness. This chain of incremental evaluations of TCB subsets combined with domain 
mechanism, which in the case of MYSEA is both logical and physical, ensures that the policies 
are enforced. 


2.2 MYSEA Concept of Operations 

Users logon to the MYSEA server using a Single Level At a Time (SLAT) client workstation. 
The commodity workstations can be on the local multilevel LAN as well as on legacy single 
level networks. Clients on the multilevel LAN must have an associated TPE. Labels for clients 
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on legacy networks are implicit, as all clients are at the level of the network itself. Each LAN- 
based client workstation and TPE is logically and physically bound together. There is exactly 
one TPE for each client workstation, and no more than one user at a time can be active on the 
given TPE. The TPE distinguishes user sessions and establishes a trusted path of communica¬ 
tion to the MYSEA server. Upon logon, the TPE establishes an identity for audit and access 
control purposes; and forwards the user’s credentials to MYSEA server. 

Next, the user uses the TPE to negotiate a session level within his security clearance range. 
Based on the security policy enforced by the MYSEA server, the session level determines the 
domain and resources he can access during that session. For example, to access resources at a 
higher classification level, the user would need to change his session level using the TPE. 

Following session level negotiation, the SLAT client-TPE pair can be used to access applications 
hosted on any MYSEA server or on a single-level network at the user’s session level. When 
activity at a particular session level is over, data created or modified on the client is stored on 
the MYSEA server, and the SLAT client state is automatically purged at the end of each session. 



Figure 2.1: Major Components in MYSEA From |1| 
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2.3 Overview of MYSEA Components and Services 

This section describes the major MYSEA components and services that comprise MYSEA’s 
design. Figure |2.1| shows the relationship between some major components and services in 
MYSEA. The major components in MYSEA include: 

1 . Clients and TPEs. Each SLAT client executes popular software applications and pro¬ 
ductivity tools such as word processors and web browsers. The TPEs controls the in¬ 
terface between the client and the MYSEA server. The TPE provides transport-layer 
security, trustworthy identification and authentication, and session establishment. 

2. MYSEA Trusted Proxy Service. Various untrusted, open source applications hosted 
on the MYSEA server provide services (e.g. web services and mail services) to clients. 
The clients cannot directly access these applications. Instead, clients invoke the services 
through a layer of proxy processes provided by MYSEA. 

3. MYSEA server Federation. The MYSEA architecture is distributed, leveraging a 
cluster of high-assurance MYSEA servers. These servers provide the locus of multilevel 
security policy enforcement and host various open source or commercial application pro¬ 
tocol servers. 

4. Dynamic Security Services. The Dynamic Security Services (DSS) manages the 
transport-layer security used to protect the trusted path between the TPE and remote 
MYSEA server. 

5. Single-Level Networks. Existing classified single level networks are connected to the 
MYSEA server providing application services to both local clients and clients on legacy 
networks. 

2.3.1 Clients and TPES. 

MYSEA clients are SLAT, and can only access and process data, at a single session level at one 
time. MYSEA clients consist of two components: commodity PCs and TPEs. Commodity PCs 
are untrusted, “thin-client” machines, running familiar commercial software and OSes. They 
are loaded with COTS products and open-source software that assist the user to perform tasks 
like word-processing, web browsing and communication. With this software, users can access 
remote applications hosted on a MYSEA server or single-level network. 

The TPEs operate largely under the direction of the remote MYSEA server and help to enforce 
MLS LAN policies. All network traffic in and out of a SLAT Client is routed through its TPE. 
The TPE ensures that a secure tunnel is established between client and TCB. This tunnel is 
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encrypted with Internet Protocol Security (IPSec) leveraging DSS services (see Section 2.3.4). 


When a remote session is established, the TPE sends a request for a service via the secure tunnel 
to the MYSEA server. The TPE acts as a Network Address Translation (NAT) device, perform¬ 
ing address translation on all traffic traversing between the client and the MYSEA server. On 
session establishment, the TPE passes the user’s credentials to the MYSEA server, which val¬ 
idates the credentials and instructs the TPE whether to allow or deny access. The user may 
use the TPE to send commands to the MYSEA server to perform session-related tasks such as 
ending the session or changing session level. 


2.3.2 MYSEA Trusted Proxy Service 

Each MYSEA server consists of a high assurance operating system that enforces the mandatory 
security policy. The high assurance kernel creates labeled protected domains and associates 
security attributes with active and passive entities exported at its interface [4], This results in 
a distinct separation of data into different security classification domains. In MYSEA, trusted 
multilevel processes control the invocation of single-level applications that interact with the user 
at his current session level. Various trusted processes are created by a daemon during system 
initialization and are responsible for the proxying and mediating network access on behalf of 
the application service, allowing it to communicate with the remote client. 

For every application service hosted on MYSEA, a Secure Session Server (Parent) (SSS-P) is 
created to respond to connection requests from clients. The SSS-P monitors the network and is 
responsible for accepting and verifying that service requests from a TPE have a valid session. 
The SSS-P spawns a proxy service (Secure Session Server (Child) (SSS-C)) for each client 
connection request. This proxy sets up the actual untrusted Application Protocol Service (APS) 
associated with the request, e.g. an untrusted Apache web service is an APS at the level of the 
remote user’s session. All subsequent communication between the client and the APS is sent 
and received via the proxy SSS-C, using one or more “MYSEA sockets”. 


2.3.3 MYSEA Server Federation 

The MYSEA server federation is a cluster of high assurance servers that provide the locus of 
multilevel security policy enforcement and hosts various open source or commercial services. 
During session establishment, the TPE forwards the user’s credentials to the MYSEA server 
hosting the Trusted Path Server (TPS) process. The TPS maintains session data including the 
user credentials, user’s clearance range and his current session level. The TPS process is respon- 
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sible for validating connection requests from the TPE, and handles TPE requests for updating 
session information (like login, change session level, run and logout), ensuring that account¬ 
ability aspects of the security policy are enforced. 

In the MYSEA server federation, a client can request an APS service hosted on a server that is 
not hosting the TPS process. A Federated Services Daemon (FSD) on each MYSEA server han¬ 
dles the internal communication required, validating the authenticity of the client and mediating 
the accesses to the requested resources according to the security policies. 

2.3.4 Dynamic Security Services 

DSS is an IPSec configuration and administrative tool that assists in the creation of secure 
tunnels between TPEs and the MYSEA server. Each DSS client resides on the TPE and connects 
to the DSS server on the MYSEA server. The DSS admin tool allows a security administrator 
to log onto the MYSEA server, set and manage the security policies associated with the IPSec 
tunnels. 


2.3.5 Single-level Networks 

Users on a legacy network can only access applications on the MYSEA server at the classifi¬ 
cation level associated with the label of the corresponding network device. Similarly, MYSEA 
clients can also access applications on the legacy networks. This allows existing applications 
on legacy networks to run unmodified and be accessible to SLAT client on the MLS LAN, with 
the MYSEA TCB mediating all accesses. 


2.4 MYSEA Security Policies 

MYSEA supports both Mandatory Access Control (MAC) and Discretionary Access Control 
(DAC) [1]. The system MAC policy is a confidentiality and integrity information flow control 
policy, where the sensitivity labels form a lattice [13]. The confidentiality and integrity policies 
are formally modeled by the BLP Jl4| ] and Biba [ 15 ] models, respectively. The high-assurance 
OS of the MYSEA server mediates access to files and other labeled system resources. Network 
connections on the MLS LAN are mediated by small, trusted processes on the MYSEA Server, 
using information about client session level. For example, a client at SECRET cannot com¬ 
municate with a client at UNCLASSIFIED, nor can a client at SECRET communicate using a 
port in use by an untrusted APS acting on behalf of a remote client at UNCLASSIFIED. Thus, 
access to network resources is impacted by checks performed by MYSEA processes, and access 
to system resources is impacted by checks performed by the underlying high-assurance OS. 
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For Discretionary Access Control (DAC), the MYSEA server allows users’ processes to modify 
permissions associated with data objects such as files via runtime functions. The permissions 
are registered with the server, and access decisions are made based on the end-users’ identities 
bounded to the processes. This policy is less critical than MAC; hence its enforcement can be 
allocated to the applications hosted on MYSEA server. Users can determine the access controls 
of APS processes to access resources such as files. The access decisions are based on the 
MYSEA user identities associated with the processes and permissions that are registered with 
MYSEA server. This policy can be modified during runtime. 


2.5 Current MYSEA Prototype 

To benchmark MYSEA, we will analyze the existing MYSEA prototype system. Here, we high¬ 
light some consequential differences between the current prototype system, and the MYSEA 
design. Our benchmark tests will be performed on the current prototype. 

2.5.1 Clients and TPEs 

There are several designs capable of satisfying the MYSEA client component. The TPE ap¬ 
plication and SLAT client could both run on a trusted separation kernel or hypervisor: each 
single-level client runs in its own partition; the TPE application runs in its own partition. Alter¬ 
natively, these components could be implemented using a small, handheld tactical device that 
connects the stateless, thin client to the MLS LAN. The current prototype system follows this 
latter strategy: the TPE mediates access to the network for a stateless client (currently a com¬ 
modity PC that must be manually purged of state whenever a session ends). The prototype TPE 
application is hosted on Linux, running on a hand-held device. 


2.5.2 MYSEA Server Federation 

Each MYSEA server is designed to run on an existing, commercial high-assurance system. 
Previous MYSEA prototypes utilized BAE XTS-400 running STOP 6, an operating system with 


CC EAL5+ certification fll6| . The current prototype uses the next generation of this product: 
STOP 7, which attained a CC EAL4+ certification [17]. 


2.5.3 Dynamic Security Services 

The security requirements of MYSEA necessitate the establishment of a secure tunnel for net¬ 
work communication between the TPE and MYSEA server. As STOP 7 lacks IPsec support, 
another mechanism for establishing a secure tunnel between the client and server is needed. 
A host-to-gateway IPSec tunnel is established between the TPE and a Server Gateway device. 
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All network traffic to the MYSEA server is routed through the Server Gateway. Together, the 
MYSEA server and Server Gateway form a single logical unit. The DSS client on both the TPE 
and Server Gateway control this IPsec tunnel, modulating the tunnel’s characteristics based on 


requests from a remote DSS admin tool. The DSS Client uses Racoon [ 18J, the user-space 
IPSec daemon, to provide the dynamic protection services. 
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CHAPTER 3: 

Objectives and High-Level Analysis 


This chapter outlines the factors that are considered during the conceptual and planning phases 
for benchmarking MYSEA as well as details about the selection of the tool used for performance 
measurement. 

We are interested in the overhead introduced by MYSEA’s design, such as those introduced 
by processes performing Mandatory Access Control (MAC) checks associated with network 
related calls. We attempt to measure components individually to determine which act as system 
bottlenecks. By isolating and measuring the behavior of individual components, rather than 
measuring the overall system or application performance, we are able to quantitatively assess 
MYSEA’s design. Figure [3TT| outlines the process flow resulting from a client’s requests to 
servers in MYSEA. 



Figure 3.1: Process flow of a client making a service request to MYSEA 


We describe the events that ensue next. 
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1. Establish IPSec Tunnel 

The DSS client on the TPE establishes an IPSec tunnel with the Server Gateway for the 
MYSEA server, forming a trusted communications path between the client and MYSEA. 
The DSS client on the TPE operates as a slave, following the DSS policy dictated by the 
MYSEA server. 

We can measure the overhead introduced as every network packet is inspected and an¬ 
alyzed according to the loaded IPSec policy. If the communication is allowable by the 
security policy, every network packet is encrypted at layer three of the TCP/IP protocol 
stack and forwarded to the MYSEA Server via the secure communications channel. 

2 . Login and Start Secure Session 

Once an IPSec tunnel has been established, the user logs on to MYSEA server using 
the TPE and establishes a session at some security level. The TPS process on MYSEA is 
responsible for identification and authentication (I&A) and must monitor for TPE connec¬ 
tion requests. There is a TPS Parent process for the MYSEA federation, which manages 
all Identification and Authentication (I&A) and login requests. The TPS Parent accepts 
login requests from MYSEA clients and spawns a TPS Child to handle trusted path com¬ 
munications for each client. When the TPS child receives data from the client, it reads 
and processes the client’s command and returns the output before waiting to receive the 
next command. Upon the successful I&A, the user at the MYSEA client negotiates a 
session within his security range to access the services hosted on the MYSEA server. 

The user can issue a “Run” or “Logout” command via the TPE. The latter terminates 
the secure session and the TPS on MYSEA takes actions to update its information about 
active sessions. Various interactions can take place during the login process, resulting in 
overhead impacting the performance of MYSEA. 

3. Service Request 

Every service request made from the client to the APS goes through a series of proxy calls 
before it is processed by the APS. The APS is the application process that responds to the 
service request, and cannot directly communicate with the client. The APS is an untrusted 
process; the SSS-C process mediates all network connections. Communications between 
the APS and SSS-C are handled through a MYSEA socket (MSKT) allocated by the TPS 
Child process. When the APS requires a socket call, the APS updates the MSKT database, 
and signals the SSS-C. The SSS-C retrieves the request from the database, responds to the 
client’s service request and inserts any retrieved data into the database. The APS is then 
notified to retrieve the requested data from the database. Each TCP connection (for UDP, 
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each packet) is checked using its source and destination, to see (in the context of active 
sessions and outstanding network use) if the information flow is allowable according to 
the MAC policy. When the APS and TPS processes are deployed on different physical 
servers, there may be additional overhead due to the communication between MYSEA 
servers, querying about the status of active sessions within the Federation. These proxy 
design decisions may impact the performance of MYSEA. 

We are interested in measuring the performance impact of the following three components: (1) 
IPSec, (2) the MYSEA trusted proxy and (3) MYSEA’s federation of servers as a result 
of the events described earlier. IPSec is related to the cost of IPSec encryption of all network 
traffic between the TPE and server gateway. Measuring the MYSEA trusted proxy allows us 
to analyze the cost of the client making a service request to MYSEA. We can measure the 
overhead associated with the proxying of system calls for making any service request. Targeting 
the MYSEA federation allows us to measure the cost of maintaining state among the servers. 

The output of a benchmark must be reproducible, consistent, unbiased and realistic, based on 
sound scientific methods and principles. The demands for a successful benchmark imply a need 
for proper scientific procedure and a need for common agreement on the metrics [19]. There¬ 
fore, the benchmarking methodology adopted in this thesis consists of a four stage process: (1) 
determine the performance indicators; (2) select or develop tools for the benchmark; (3) config¬ 
ure the test environment and; (4) execute benchmarks and analyze output. We describe stages 
(1) and (2) in this chapter, stage (3) in Chapter |4] and stage (4) in Chapter [5j 


3.1 Benchmarking Methodology 

Benchmarking is the process of comparing business processes, services or products against one 
another. It allows organizations to understand quantitatively the performance of the targets, 
possibly for the purpose of improving their characteristics. Benchmarking of an IT system is 
the process of running a specific program or workload on a specific machine or system and 
measuring the resulting performance [20]. 


There are a wide variety of approaches to benchmarking because different approaches are neces¬ 
sary to evaluate individual aspects of a system. For example, synthetic benchmarks load or stress 
test some specific targeted components in the system; application benchmarks measure perfor¬ 
mance of the system as it accomplishes some real-world task (like running a web server) under 


some synthetic load. Benchmarks can be divided into the following categories [T9| [211 [22]: 
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• Application benchmarks, also known as macro benchmarks, take an application and 
measure the time required to complete some tasks. They measure end-to-end perfor¬ 
mance on a specific workload, assessing the performance experienced by end-users. For 
example, the library generator ATLAS (23) uses application benchmarking to build linear 
algebra libraries automatically tuned for the target processor. An evaluation of the Denali 
Isolation kernel (24) made use of web server benchmarks to compare that platform with 
BSD. The SPEC CPU benchmark suite measures several applications and combines the 
results to rank overall performance. 

• Micro-benchmarks measure the performance of individual operations in isolation. Bench¬ 
marks intended to measure the performance of kernel primitives; system calls and the 


network stack are commonly measured using micro-benchmarks. Ousterhout [25] de¬ 
veloped benchmarks to isolate a number of kernel primitives providing measurements of 
performance, independent of the machine’s platform. Micro-benchmarks were also used 
to quantify the performance of the Denali Isolation Kernel’s primitive operations (24) . 
Network micro benchmarks measure the bandwidth, throughput and network latency ex¬ 
perienced by the system, irrespective of application. Such benchmarks provide insight 
into low-level network stack performance; in particular the latency associated with open¬ 
ing a socket connection. A performance characterization of the Chelsio T110 10-Gigabit 
Ethernet (26) adaptor makes use of micro-benchmarks to measure the single and multiple 
connections of point-to-point latency and unidirectional throughput. 

• Other benchmarks target the hardware used by the system, or the end-users of the sys¬ 
tem. Benchmarking I/O performance provides a measurement of the cost of retrieving or 
storing data using some particular hardware. It may measure sequential and streaming I/O 
bandwidth for single processes, or random I/O latency for multiple processes (including 
the time taken to create, write and read a number of files in a file system). 

Subjecting real-world users to the system provides realistic performance measures. These 
user tests produce results that are not always comparable across systems or versions, as 
these tests cannot be easily repeated. 


3.2 Performance Indicators 

Different tools measure different performance indicators such as processor speed, memory la¬ 
tency, network throughput, bandwidth and socket connection latency. Chapter [2] describes the 
MYSEA process structure and their relationships. For a user to access an APS service hosted on 
MYSEA, secure communication is established between the MYSEA client and MYSEA server 
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using IPSec encryption. Every network packet is inspected and encrypted at the TPE before 
it is transmitted to the server gateway. At the server gateway, network packets are decrypted 
and forwarded to the MYSEA server for processing. Also, during the initial setup of APS com¬ 
munication, the SSS-P listens on the network for requests and spawns a proxy to handle the 
connection. The proxy acts as an intermediary between the APS and network to handle the 
request. All traffic is inspected and proxied within MYSEA before the APS can receive it. In 
addition, the TPS process that handles all I&A can be deployed on a different server from those 
hosting MYSEA’s APS services. If a service request is made to an APS hosted on a server other 
than the TPS, internal communication between the servers is required to assert the identity and 
authenticity of the request. The following two KPIs are identified to measure the performance 
of MYSEA. First, we can measure the TCP and UDP throughput transfer rates of the IPSec 
encryption, MYSEA proxy and MYSEA federation. Second, we can also measure network 
latency, the time taken for data to be transmitted from one point to another. In MYSEA, one 
could measure the latency associated with opening and closing a network socket, reflecting the 
initial costs of the setup for a connection between the client and server. We can also measure 
the latency associated with the time taken for a request and response transaction to complete. 

3.3 Benchmarking Tools for Use in MYSEA 

There are various benchmarking tools available to measure the performance of systems that 
provide networked services such as MYSEA. Examples include Apache Jmeter, LMbench and 
Netperf. These benchmark tools automate the process of running performance tests for the 
targeted system. These tools are intended to produce repeatable results, allowing for consistent 
analysis of data to compare system performance. 

Any benchmark tools used in the context of MYSEA must satisfy certain operational require¬ 
ments. The current MYSEA prototype runs on the STOP operating system, a proprietary high 
assurance operating system that is binary compatible with Linux. Any application service 
hosted on MYSEA needs to be modified to run in a specific inetd mode, integrated with MY¬ 
SEA processes. Therefore, source code for the tool selected must be available, and compatible 
in the MYSEA environment. 

The tool selected must be able to measure specific targets of the MYSEA system. We desire to 
characterize each MYSEA component to analyze each feature of MYSEA’s design. In particu¬ 
lar, an end-to-end application benchmark tool will not be able to produce such measurements, 
as it characterizes the system only in terms of its performance with a specific application. For 
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example, MYSEA currently supports a variety of application services (web services, VoIP, etc.), 
and the benchmark tool selected must not characterize the performance of the application ser¬ 
vice. Additionally, as MYSEA is an experimental system without actual users, we are unable to 
develop workloads representative of its use in a realistic operating environment. The following 
sections described the benchmark tools considered. 


3.3.1 Jmeter 


Apache Jmeter (TM) [27] is an open-source web server benchmark written in Java designed to 
test functional behavior and measure performance. It can be used to test performance on static 
and dynamic web resources, and perform load testing on the server. It is extensible and allows 
the development of plugins that provides extensible testing capabilities. 


Jmeter can be configured to benchmark the performance of the existing web services on MYSEA 
at the client. This would eliminate the need to modify or adapt the tool to run on MYSEA. How¬ 
ever, Jmeter is unable to produce results that measure isolated components of MYSEA, for ex¬ 
ample the cost of proxying specific network calls. This tool would characterize the performance 
costs of MYSEA’s TCB, combined with the performance characteristics of the untrusted, third 
party web service, rather than characterizing the MYSEA system performance in isolation. 


3.3.2 Netperf 

Netperf [ 28 ] is an open-source benchmark tool written in C that is designed to characterize gen¬ 
eral network performance experienced in client-server settings. The tool is split into two com¬ 
ponents: netperf and netserver. The netperf tool is a client that requests tests to be performed, 
while the netserver tool is a daemon process that resides on the system under test and responds 
to requests. Netserver can run as either as a daemon, or as an inetd service. Netperf provides 
tests for both unidirectional and bidirectional throughput and end-to-end latency. It can be used 
to measure various aspects of networking performance with primary focus on unidirectional 
data transfer and request/response performance using TCP or UDP socket interfaces. 


3.3.3 LMbench 

LMbench |22j f29fl is a suite of micro-benchmarks written in C designed to measure latency 
and bandwidth performance of system operations like data movement among the processor 
and memory, network, file system and disk. LMbench was developed to identify and evaluate 
system performance bottlenecks, and is portable across a wide set of Unix systems. LMbench is 
open-source and binary compatible with Linux. It can also be used to measure specific targeted 
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components and servers. However, LMbench lacks comprehensive documentation: its manual 
pages lack detail, and the documentation does not describe how each specific test measures its 
target. The lack of documentation hinders the porting of LMbench source code to work in the 
MYSEA environment. 

3.3.4 SPEC 

Standard Performance Evaluation Corporation (SPEC) is a non-profit corporation that develops 
and maintains a standardized set of relevant benchmarks for measuring the performance of IT 
systems. Organizations can measure the performance of their products or services using the 
SPEC benchmark suites. The results can be submitted to SPEC for validation, creating an 
independently validated benchmark. This provides a fair comparison between similar products 
and services. SPEC suites of benchmarks are written in C, Java or Fortran. However the source 
code cannot be modified under its licensing rules to prevent the optimization of benchmarks to 
skew the performance results positively, affecting fair comparison. This limits the portability 
requirement of the benchmark tool. 
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CHAPTER 4: 
Methodology 


This chapter describes the test environment, configurations and test plan used to benchmark 
selected key performance indicators in MYSEA. The tests are designed to produce results from 
which unbiased comparisons can be drawn. The benchmarks have been documented so that 
they can be repeated with the same experimental setup. 

The test environment consists of the physical setup, which includes the servers, network routers, 
switches, and their interconnection. The software configuration used in the benchmark tests 
consists of the server operating systems, client operating systems and the MYSEA components 
and services. 

We can configure different setups that allow for the measurement of targeted components (see 
Chapter[3]> of the MYSEA server. The setups are formed by combining permutations of the three 
components, resulting in a total of eight different setups. We compare the eight different setups 
against one another. The comparisons of different setups are referred to as Test Configuration 
(TC)s where each TC measures the performance of the components and services in isolation. 

We can execute a suite of benchmark tests on each TC, measuring the different aspects of perfor¬ 
mance of the isolated components and services. Netperf client executes the following five tests: 
(1) TCP STREAM, (2) TCP_CRR, (3) TCP_CC, (4) TCP_RR and (5) UDP_STREAM. 

We describe the tool selected and modifications that enable the benchmarking of MYSEA in 
Section |4.1[ We describe the test environment for the benchmarks in Section |4.2[ the various 
TCs in Section 1431 and the suite of benchmark tests executed on MYSEA server in Section l4~4l 
Section |43] describes the test plan for performing the benchmarks. 

4.1 Tools and Instrumentation 

Netperf version 2.5.0 is selected as the tool used to benchmark MYSEA. There are three reasons 
behind the selection of netperf as the benchmark tool. First, netperf is open-source and, second, 
it is written in the C programming language. The necessary modifications to the netperf source 
code can be made to enable it to run on the current MYSEA prototype. Third, netperf has 
comprehensive documentation that describes in detail how each test performs the benchmark 
and what the test is measuring. 
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To benchmark MYSEA, we deploy the netperf client on the TPE, and netserver on the MYSEA 
server. The netperf client runs a suite of benchmark tests, making requests to the netserver 
daemon. Netserver responds to these requests, measuring latency and throughput performance 
and returns the results to netperf. 

Netserver has two modes of operations: inetd and daemon. The inetd mode is used in TCs using 
MYSEA, since all MYSEA processes are started by a trusted inetd process. The daemon mode 
is used in TCs where MYSEA is not being measured, and it is not natural to start the process 
via inetd. 

In the inetd mode of operation, the MYSEA trusted inetd process SSS-P spawns the netserver 
process when a request is received on the netserver service’s listening port. Netserver has been 
modified to act as an untrusted MYSEA service and thus uses the MYSEA proxy service. This 
allows us to measure the proxy service performance. Refer to Appendix [A] for more details on 
the modifications to the netserver source code. 

In the daemon mode of operation, one starts the netserver process manually before testing. The 
netserver daemon process monitors the network, responding to requests from the netperf client. 
This mode of operation does not require any modification to the netserver source code. 

Appendix[B]provides details on how the netperf client and netserver server process are installed, 
and how the benchmark tests are executed. 
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4.2 Test Environment 


MYSEA 1 



Cisco Catalyst 3560G 


Figure 4.1: The MYSEA benchmark test environment (network configuration and hardware). 


Figure |4.1| shows the configuration and topology of the multilevel LAN used during bench¬ 
marking. All network connections are one gigabit per sec (lGbps) to reduce the possibility of 
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Table 4.1: Configurations of TPE, server gateway and MYSEA server 


Server Name 

Configuration 

TPE 

512MB RAM, VMWare ESXi v4.1.0 

Fedora 7 linux kernel 2.6.23, IPSec Tools vO.6.6 

Server Gateway 

3GB RAM, VMWare ESXi v4.1.0 

Fedora, IPSec Tools vO.6.6 

MYSEA server 

2GB RAM, VMWare ESXi v4.1.0 

STOP v7.3.1 


limited network bandwidth affecting the results of our benchmarks. The configurations of the 


TPE, server gateway and MYSEA servers are provided in Table 4.1 


4.3 Test Configuration 

Different TCs are designed to measure and compare the performance the MYSEA components 
and services in isolation. Figure |4.2| compares all possible test configurations. The setups are 
formed by the combination of three pairs of scenarios: “M”/“M-” and “F7“F-”. Each 

scenario is described next. 

“I” represents the scenario when IPSec is used to encrypt all network traffic between the TPE 
and the MYSEA server gateway. “I-” represents the scenario when no network traffic is en¬ 
crypted in the IPSec tunnel. This is achieved by flushing all IPSec rules from Racoon. There 
are subtle differences between setups A5, B6, C7 and D8, although the TCs indicate that they 
measure the performance of IPSec encryption. A5 and B6 measure the performance of IPSec 
encryption with netserver spawned in inetd mode in a MYSEA environment where netserver is 
accessed via MYSEA trusted proxy calls. C7 or D8 is deployed as a daemon on STOP server. 

“M” represents the scenario when the netperf request undergoes a series of proxy calls in 
MYSEA before reaching the netserver process, with netserver process spawned by the MYSEA 
trusted inetd process. A secure session is first established between the TPE and the MYSEA 
server before netperf client executes the suite of benchmark tests. “M-” represents the sce¬ 
nario when the netserver process is running as a daemon on STOP with no MYSEA processes 
running. 
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“F” represents the scenario when the TPS process and the netserver process are deployed on 
different physical servers in the MYSEA federation. This results in additional internal network 
communication between the servers when netperf client executes the tests. “F-” represents the 
scenario when the TPS process and our benchmark tool are deployed on the same physical 
server. 

For example, “A5” compares the two test configurations “I M F” and “I-M F”. This test config¬ 
uration measures the overhead associated with IPSec (“I” vs in the configuration where 
MYSEA services are being used (“M”) on servers that require internal, federated requests (“F”). 
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Setups 



Figure 4.2: Figure showing the comparisons of different test configurations. Test cases considered in 
this thesis are underlined 
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4.4 Types of Tests 

Netperf is capable of running different types of tests to measure different aspects of performance 
of the targeted system. Five different tests are executed on the setups specified in Figure |4.2[ 
TCP_STREAM, TCP_CRR, TCP_CC and TCP_RR, UDP_STREAM. These five tests form 
the suite of benchmark tests. (30]] provides additional descriptions of the various tests. Each of 
these will be discussed in their respective sections below. 

We made use of the “OMNI” test in netperf to run the various tests. Previous netperf versions 
included different source code to perform different tests. The “OMNI” test in netperf is a 
consolidation of all available test functions into a single test. Different flags allow different 
options to be selected to enable the execution of specific test functions. There are two types of 
flags: global and test-specific. “Global” flags (see Table |4.3| ) provide options that apply to all 
benchmark tests. Test-specific flags are described in their respective subsection. 

Table 4.3: An explanation of the global options used in netperf 


Flags 

Value 

Description 

-H 

hostname 

This option sets the hostname or IP address of the targeted sys¬ 
tem, where netserver daemon process is deployed. This option 
allows us to measure the performance of the two different physi¬ 
cal servers under the scenarios: “F” and “F-”. 

-P 

0 

A value of “0” for this flag disables the display of the test banner. 
The test banner was disabled in all of our experiments because we 
executed the benchmark tests multiple times in succession and the 
test banners are redundant and unnecessarily cluttered the output 

-P 

port 

This option sets the port number for netperf client to use to con¬ 
nect to the netserver daemon process. 

-0 

filename 

This option allows the netperf client to read in a file containing 
comma-separated header values. These headers correspond to the 
different output values we want to record in our results. Exam¬ 
ples of these headers include “THROUGHPUT”, “THROUGH- 
PUTJJNITS”, “PROTOCOL” and “ELAPSED_TIME”. 


4.4.1 TCP_STREAM 

MYSEA clients access various services (i.e., web and mail) hosted on the MYSEA server us¬ 
ing Transmission Control Protocol (TCP)-based protocols. The performance of MYSEA in 
handling TCP packets is important to the experience of the client, in terms of the latency and 
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throughput observed by the client. The TCP_STREAM test is designed to measure the per¬ 
formance of the targeted component as it processes TCP data packets. The test provides the 
throughput in bits per second and accounts for both the time required to push data across the 
network and the time to process it on the targeted component. This test is unidirectional and the 
time spent establishing the connection is not included in the throughput calculation. Table |4.4| 
shows the test-specific options used in netperf for the test: TCP_STREAM. 

Table 4.4: TCP_STREAM test-specific options 


Flags 

Value 

Description 

-d 

send 

This option sets the direction of the test relative to the netperf 
process. A value of “send” causes netperf to send data to the 
netserver daemon process, resulting in a unidirectional test. 

-T 

TCP 

This option sets “TCP” to be the protocol used for the test. 


4.4.2 TCP_CRR 

The TCP_CRR test combines TCP_CC and TCP_RR and measures the performance of the tar¬ 
geted component in handling TCP connections. This test allows us to measure the performance 
overhead in transactions per second of the targeted component in completing the entire TCP pro¬ 
cess: establishing a connection via the three-way handshake, transferring one byte of request, 
receiving one byte of response and tearing-down the connection. We can then obtain the latency 
of one complete transaction in seconds by inverting the results. This test is designed to measure 
the end-to-end latency experienced by a client when making a service request. Table |4.5| shows 
the test-specific options used in netperf for the test: TCP_CRR. 

Table 4.5: TCP_CRR test-specific options 


Flags 

Value 

Description 

-c 

This option explicitly instructs netperf to include connection es¬ 
tablishment and teardown in its measurement. 

-d 

send 1 recv 

This option sets the direction of the test relative to the netperf pro¬ 
cess. A value of “send” causes netperf to send data to netserver 
daemon process, while the value of “recv” causes netserver dae¬ 
mon process to send data back to netperf. This results in a bi¬ 
directional receive and response test. 

-T 

TCP 

This option sets “TCP” to be the protocol used for the test. 
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4.4.3 TCP_CC 

The TCP_CC test isolates the speed at which connections can be opened and closed between 
the target and client using the TCP protocol. A three-way handshake establishes a reliable 
connection between the sender and receiver. The test is performed in a synchronous manner, 
with no data transferred between the two entities. This test continuously opens and closes 
connections between netperf and netserver, tracking the number of connections created. This 
test provides the latency in transactions per second, a measurement of the performance of the 
targeted component in establishing and tearing-down TCP connections. 

The MYSEA trusted proxy introduces additional overhead for every new client connection. For 
each new client connection, a proxy SSS-C process is spawned to handle the request. This test 
measures the impact of spawning the additional process to handle the client request. Table |4.6| 
shows the test-specific options used in netperf for the test: TCP_CC. This test measures the 
latency associated with the opening and closing of a transaction. 


Table 4.6: TCP CC test-specific options 


Flags 

Value 

Description 

-c 

This option explicitly instructs netperf to include connection es¬ 
tablishment and teardown in its measurement. 

-T 

TCP 

This option sets “TCP” to be the protocol used for the test. 


4.4.4 TCP_RR 

The TCP_RR test is a request and response test using the TCP protocol, where each request and 
response packet is one byte in size. It is bi-directional and measures the number of complete 
transactions per unit time that can be exchanged between two entities. This test is indepen¬ 
dent of the time taken to establish the initial TCP socket connection and measures the latency 
associated with the transfer of a byte of request and response request between the client and 


server. Table |4.7| shows the test-specific options used in netperf for the test: TCP_RR. This test 
is designed to measure the speed for uploading/downloading large files to MYSEA server (e.g., 
the ftp server), where the initial cost of setting up a connection is less significant. 


4.4.5 UDP_STREAM 

UDP is used as the underlying transport protocol for VoIP, and some other services provided 
by MYSEA. The UDP_STREAM test is similar to the TCP__STREAM test except that UDP 
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Table 4.7: TCP RR test-specific options 


Flags 

Value 

Description 

-d 

send 1 recv 

This option sets the direction of the test relative to the netperf 
process. A value of “send” causes netperf to send data to net- 
server daemon process, whereas the value of “recv” causes net- 
server daemon process to send data back to netperf. Used in com¬ 
bination, this results in a bi-directional receive and response test. 

-T 

TCP 

This option sets “TCP” to be the protocol used for the test. 


is used as the mode of transport rather than TCP. This test provides the throughput in bits per 
second and measures the performance of the targeted components as they process UDP data 
packets. Table |4.8| shows the test-specific options used in netperf for the test: UDPJSTREAM. 

Table 4.8: UDP STREAM test-specific options 


Flags 

Value 

Description 

-d 

send 

This option sets the direction of the test relative to the netperf 
process. A value of “send” causes netperf to send data to netserver 
daemon process, resulting in a unidirectional test. 

-T 

UDP 

This option sets “UDP” to be the protocol used for the test. 


4.5 Test Plan 

Netperf client runs the suite of benchmark tests on the following six setups: “I M-F”, “I M- 
F-”, “I-M-F”, “I-M-F-”, “I-M F” and “I-M F-” (See Figure |43J). Each benchmark test was 
executed sequentially for ten seconds (netperf default), and each test is repeated for one hundred 
trial tests. We conducted experiments in the initial laboratory setup and repeated the trials with 
different numbers of times: 10, 50, 100, 1000, 10000. One hundred (100) trials was selected 
as it optimizes the amount of time required for the completion of the entire suite of benchmark 
tests while achieving significant population samples with low variances. Before any execution 
of benchmark tests, the test environment was restarted and prepared for the respective setup. 
Steps included: recompilation of the netserver source code for the respective scenario (“I-M F”, 
“I-M F-”, “I-M F”, “I-M F-”, “I M-F” and “I M-F-”), configuration of IPSec to setup/teardown 
the secure tunnel, and deployment of netserver on the appropriate server. The results of the tests 
were collected at the end of experiment. 
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For the TCP_CC, TCP_CRR and TCP_RR benchmark tests, a two-minute delay wait period 
between successive tests was introduced to address the following problem. During initial trial 
experiments, abnormal results were obtained when these tests were executed. The latency of 
certain trials was several times lower than the rest of the trials. After several experiments, it 
was determined to be an issue associated with a TIME_WAIT reuse in the establishment of a 
socket connection. The latency drops dramatically when the test reuses a connection that is 
in the TIME_WAIT state. This results in a non-trivial delay in connection establishment. The 
two-minute delay wait period forced the TIME_WAIT to expire. 
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CHAPTER 5: 
Testing and Analysis 


This chapter discusses the execution of the benchmark tests and provides analysis of the results. 
Section |5T| discusses issues related to the collection of data and Section |5.2| provides a summary 
of the results. 


5.1 Overview 

Chapter [3] and Chapter [4] discuss the methodology for designing a meaningful benchmark to 
measure the performance of targeted components in MYSEA. The goal is to achieve soundness : 
developing confidence that the results we derive from our measurements are well justified, with 
a solid understanding of the strengths and limitations of the measurement process on which we 
base our results [31J. We need to calibrate the test results by examining outliers and testing for 
consistencies. This allows us to reliably reproduce analysis results by imposing a systematic 
structure on the analysis process. 

The “OMNI” test in netperf allows the specification of the type of output values to collect for 
the test. These “selectors” determine the data to be reported. Table |5 .1 1 highlights the important 
selectors used. |30} provides the detailed documentation for the “selectors”. 


Table 5.1: Netperf "OMNI” test selectors used during benchmarking MYSEA 


Selector Name 

Description 

Examples 

PROTOCOL 

Protocol used in the bench¬ 
marks. 

UDP, TCP 

DIRECTION 

Direction of the data flows 

relative to the netperf process. 

send 1 recv: bidirectional re¬ 
quest and response 
send: unidirectional request 

THROUGHPUT 

Throughput for the test. 

350.0234 

THROUGHTPUTJJNITS 

Units for the throughput 

10 3 bits/s or 10 6 bits/s 
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5.2 Summary of Results 


We compare each test scenario, relative to the scenario in which netserver is deployed on STOP 
without the network being protected with IPSec, treating this as a “base case”. As expected, 
we see that the base case shows the best performance, i.e., highest throughput and lowest la¬ 
tency (see Figure |5.1[ |5.2| and |5.3| ). For TCP stream performance, IPSec encryption results in 
a 60% decrease in throughput, whereas MYSEA processes result in between 21-24% decrease 
in throughput. For socket related performance, IPSec encryption results in between 24-30% 
increase in latency, whereas MYSEA processes result in between 93-143% increase in latency. 
These observations will be described more in their respective sections. 


Our UDP results are inconclusive. For setups without IPSec encryption, the throughput values 
obtained are lower than when IPSec encryption tunnel is used. This is counter-intuitive and 
possibly the result of packet loss during testing. We refer the interested reader to Appendix [G] 
for the status of our UDP test results. We leave further UDP testing as future work. 


5.2.1 Overhead associated with IPSec 

IPSec encryption reduces unidirectional TCP transfer throughput significantly. The perfor¬ 
mance consequences of IPSec in an architecture similar to MYSEA, with server gateway and 
IPSec encryption of traffic between TPE and server, is a decrease in unidirectional transfer 
speeds, by about 60% (see Table |5.3| ). TCP connect latency increases an average of 24-30%, 
reflected in the TCP_CC, TCP_RR and TCP_CRR tests. For the details of each test, see Ap¬ 
pendix [D| 

The performance of IPSec has been well studied. Shue et al [32 ] report performance overhead 
of 32-60% for Internet Key Exchange (IKE) based IPSec (Racoon uses IKE based IPSec) and 
provides a comprehensive study on the performance overhead of IPSec. This is similar to our 
results of 60% decrease in TCP throughput and socket related performance overhead range of 
24-30%. 


5.2.2 Overhead associated with the MYSEA Trusted Proxy 

We observe a 21-25% decrease in throughput for unidirectional TCP traffic, relative to the base 
case (see Table |5.4| >, when the trusted MYSEA proxy is used for network traffic (for details, see 
Appendix [E]). We believe this decrease can be attributed to MYSEA’s proxy system call design, 
in which the client and APS communicate via a proxy process (SSS-C) mediating all accesses, 
inspecting and routing TCP network packets internally before reaching the APS. 
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Table 5.2: 

IHb 


Summary of the average overhead witnessed during TCP benchmarks (Figure 


5.1 


5.2 


TCP STREAM Throughput 


Scenario 

Mean (10 b bits/s) 

% decrease 

MYSEA 

I-MF 

I-M F- 

274.95 

270.57 

21.67% 

24.67% 

Base 

I-M-F 

I-M-F- 

351.00 

359.16 

-% 

-% 

IPSec 

I M-F 

I M-F- 

139.70 

144.51 

60.20% 

59.76% 


TCP CC: Latency (ms/Trans) 


Scenario 

Mean (Trans/s) 

Latency 

% increase 

MYSEA 

I-MF 

I-M F- 

529.54 

674.70 

1.888 

1.482 

143.30% 

93.22% 

Base 

I-MF 

I-M F- 

1288.58 

1304.01 

0.776 

0.767 

-% 

-% 

IPSec 

I-MF 

I-M F- 

993.89 

1020.16 

1.006 

0.980 

29.64% 

27.77% 


TCP RR: Latency (ms/Trans) 


Scenario 

Mean (Trans/s) 

Latency 

% increase 

MYSEA 

I-MF 

I-M F- 

2318.23 

2338.82 

0.431 

0.428 

20.40% 

19.55% 

Base 

I-MF 

I-M F- 

2791.03 

2792.37 

0.358 

0.358 

-% 

-% 

IPSec 

I-MF 

I-M F- 

2148.25 

2243.02 

0.465 

0.446 

29.89% 

24.58% 


The latency observed when the establishment of a new connection is involved is anywhere 


between 90-150%, relative to the base case (see Table |5.4| ), whereas TCP_RR results in a 
smaller increase in latency at about 20%, similar to the TCP_STREAM performance overhead. 
TCP_RR is similar to TCP_STREAM, with the former being a bidirectional test. The TCP_RR 
and TCP_STREAM results further conclude that MYSEA processes increase TCP performance 
overhead by about 20% as a consequence of its proxy design to route all network packets via its 
respective SSS-C process. 


However, performance overhead increases significantly to 90-150% when the establishment of 
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Table 5.3: Summary of IPSec results 


TCP_STREAM Throughput 


Scenario 

Mean (10 b bits/s) 

% decrease 

F- 

I-M-F- 

IM-F- 

359.16 

144.51 

60.00% 

F 

I-M-F 

I M-F 

351.00 

139.70 

60.20% 


TCP_CC: Latency (ms/Trans) 


Scenario 

Mean (Trans/s) 

Latency 

% increase 

F- 

I-M-F- 

I M-F- 

1304.01 

1020.16 

0.767 

0.980 

27.77% 

F 

I-M-F 

I M-F 

1288.58 

993.89 

0.776 

1.006 

29.64% 


TCP_RR: Latency (ms/Trans) 


Scenario 

Mean (Trans/s) 

Latency 

% increase 

F- 

I-M-F- 

I M-F- 

2792.37 

2243.02 

0.358 

0.446 

24.58% 

F 

I-M-F 

I M-F 

2791.03 

2148.25 

0.358 

0.465 

29.89% 


TCP_CRR: Latency (ms/Trans) 


Scenario 

Mean (Trans/s) 

Latency 

% increase 

F- 

I-M-F- 

I M-F- 

1183.12 

903.96 

0.845 

1.106 

30.89% 

F 

I-M-F 

I M-F 

1152.27 

889.10 

0.868 

1.125 

29.61% 


a new connection is involved. We believe this can be attributed to the creation of a new SSS-C 
process for every new TCP connection, resulting in significant performance overhead. 

5.2.3 MYSEA Federation Overhead 

When a MYSEA service (APS) and the TPS process are deployed on different servers, the 
APS’s proxy causes a Federated request to fetch information about the client’s current level from 
the MYSEA server hosting the TPS. In practice, these security checks occur per connection (for 
UDP, per packet). In our testing, the netserver process acts as an APS, allowing us to interpret 
the overhead associated with the intra-Federation communication. 
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Table 5.4: Summary of MYSEA results 


TCP_STREAM Throughput 


Scenario 

Mean (10P bit s/s) 

% decrease 

F- 

I-M-F- 

I-M F- 

359.16 

270.57 

24.67% 

F 

I-M-F 

IM-F 

351.00 

274.95 

21.67% 


TCP_CC: Latency (ms/Trans) 


Scenario 

Mean (Trans/s) 

Latency 

% increase 

F- 

I-M-F- 

I M-F- 

1304.01 

674.70 

0.767 

1.482 

93.22% 

F 

I-M-F 

I M-F 

1288.58 

529.54 

0.776 

1.888 

143.30% 


TCP_RR: Latency (ms/Trans) 


Scenario 

Mean (Trans/s) 

Latency 

% increase 

F- 

I-M-F- 

I M-F- 

2792.37 

2338.82 

0.358 

0.428 

19.55% 

F 

I-M-F 

I M-F 

2791.03 

2318.23 

0.358 

0.431 

20.39% 


TCP_CRR: Latency (ms/Trans) 


Scenario 

Mean (Trans/s) 

Latency 

% increase 

F- 

I-M-F- 

I M-F- 

1183.12 

575.88 

0.845 

1.736 

105.44% 

F 

I-M-F 

I M-F 

1152.27 

467.30 

0.868 

2.140 

146.54% 


For TCP services, we observe a greater latency when the APS and TPS are deployed on different 
servers. From the results of the TCP_CC and TCP_CRR tests, setup “F” shows a significant 
increase in latency of about 50% compared to setup “F-”. We believe this can be attributed 
to the additional performance overhead associated with the MYSEA processes communicating 
internally when the APS and TPS are on different servers. 
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Figure 5.1: Boxplot showing the summary of TCP_STREAM performance for the six setups 
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Figure 5.2: Boxplot showing the summary of TCP_CC socket performance for the six setups 
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Figure 5.3: Boxplot showing the summary of TCP RR request and response performance for the six 
setups 
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CHAPTER 6: 
Conclusion 


We analyzed the architecture of the MYSEA framework, and identified the major components 
and services that create a high assurance MLS environment. Overall, we observe that the MY¬ 
SEA trusted proxy architecture incurs significant overhead (impacting throughput and latency 
for network connections). On average, the latency for new connections more than doubles 
(about 118%) compared to MYSEA’s underlying platform. These costs are associated with 
connection setup, i.e. starting a socket proxy on behalf of the user. Once a connection is es¬ 
tablished, we observe an average throughput decrease of about 23% for unidirectional TCP 
streams, associated with the cost of proxying a socket write() request. In a Federated setting, 
the way state is communicated among servers alone is associated with an increase in latency of 
about 50%. 

6.1 Recommendations for Future Work 

This thesis performed a high level analysis on the performance of MYSEA, focusing on the 
comparison of the mean latency values and throughput. Further detailed mathematical analysis 
can be performed on the data, finding other trends for the current set of data. The UDP results 
are not conclusive, and future work can look into the possible causes and mitigations to study 
the performance overhead of UDP packets. Future work can analyze other components and 
services in MYSEA, such as the performance of the TPE and server gateway. Future work can 
explore recommendations for improving the performance of the bottlenecks identified. Future 
work can also measure the performance of MYSEA in setups “I M F-” and “I M F”, which are 
closer comparisons to the operational environment that MYSEA is deployed in. 
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APPENDIX A: 
Netserver Modifications 


This appendix describes the modifications to netserver to operate in the MYSEA environment. 

1. Include MYSEA headers 

In the header file: netlib.h, insert the following include statement. 

#include "mysea_portability.h" 

2. Insert flags for different compilation modes 

Flags are used for the compilation of netserver in its two different modes of operation: 
inetd and daemon, inetd mode is used in the MYSEA environment, under scenario “M", 
while daemon mode is used in STOP, under scenario “M-". We need to compile the 
netserver source code with support for MYSEA processes in scenario “M". Any modifi¬ 
cations made to the source code to operate in MYSEA environment use flags inserted on 
the command line to obtain the appropriate compilation. 


#ifdef USE_M_SQCKET 

#include "mysea_portability. h" 

#endif 


Figure A.l: Insertion of MYSEA header with flags to compile for MYSEA support 

3. Insert necessary code 

Additional code and functions are added to netserver to integrate with MYSEA processes. 
The following functions and headers are added to the source code: 


#ifdef USE_M_SQCKET 
int my_fd : = -1; 
#endif 


void exit_mysea {void) 

{ 

#ifdef USE_M_SOCKET 
mport_exit(0); 
#endif // USE_M_50CKET 
exiti0); 

> 


Figure A.2: Addition of headers to source code in file: netserver.c 
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void 

set mysea server sock(int enskt fd) { 

#ifdef WIN32 

server sock = ( SOCKET)GetStdHandlef STD 
#elif !definedf VMS) 

if (server_sock != INVALID_SOCKET) { 
printfC'yo, Iz ain't invalid!\n" ] ; 
exit(l) ; 

.INPUT_ HANDLE) : 

.r 

server_sock = msk t_Tcf ; 

#else 

printf ( H \tALL ELSE\n M ) ; 

if (( server_sock = *nskt_fd) == INVALID_SQCKET) { 
f printf f sttferr, 

"%s: failed to grab aux server socket: *fcs (errno %s)\n" p 
_FUNCTION_ _ p 

strerrorierrno)„ 
errno); 
fflushCstderr ) : 
exit(l); 

} 

#endif 

> 



Figure A.3: Addition of functions to source code in file: netserver.c, function: main 
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int _cdecl| 

main(int argc r char *argv[]) { 

#ifdef USE_H_SOCKET 
int fnysea_exit = 0; 
int result: 

// Initialize underlying MYSEA modules 
result = mport_init ( APS_M0DE r ,a netserver_log M , NULL): 
if ( result != MP0RT_0K) { 
result = -1; 

printf ("fnport_init error\n M ); 
mysea_exit = 1; 

>else{ // find the MSKT fd 

result = mport_get_fd(&my_fd) : 
if [ result ! = MPORT_OK){ 
my_fd = -1: 

printf ("fnport_get_fd error\n") ; 
mysea_exit = 1; 

}else{ 

result = mport_get_file_stream(Sphere) : 
if [result != HPORT_0K)-[ 

printf [ M mport_get_file_stream error") ; 
mysea_exit = 1; 

}else{ 

f printf[uhere, "Using HSKT fd [%d]\n" p my_fd) : 

> 

> 

> 

if [mysea_exit ) { 

printf ("Initizalition error, exiting"): 
mport_exit ( 1 ): 

> 

atexit(&exit_mysea) : 

/^result stores the mskt_fd */ 

#endif // U5E_M_5QCKET 
#ifdef WIN32 


Figure A.4: Modifications to source code in file: netserver.c, function: main 
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#ifdef USE_M_SOCKET 

set_mysea_server_sock(my_f d) ; 

/^open_debug_file() ;+/ 

process_requests( ) ; 

#else 

if (child) { 

/* we are the child of either an inetd or parent netserver via 
spawning (Windows) rather than forkding. if we were forkOed 
we would not be coming through this way. set_server_sock() must 
be called before open_debug_file() or there is a chance that 
we'll toast the descriptor when we do not wish it. */ 
set_server_sock( ); 
open_debug_file( ); 
process_requests( ) ; 

> 

else if (daemon_parent ) { 

/* we are the parent daemonized netserver 
process. accept_connections[) will decide if we want to spawn a 
child process 
accept_connections( ) ; 

} 

else | 

/* we are the top netserver process, so we have to create the 
listen endpoint(s) and decide if we want to daemonize */ 
setup_listens(local_host_name r listen_port , local_add ress_family ); 
if (want_daemonize) I 
daemonize () ; 

> 

accept_connections[ ) ; 

>i 

#endif 


Figure A.5: Modifications to source code in file: netserver.c, function: main 
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APPENDIX B: 

Installing and Running the Benchmark Tool 


Two files are necessary for the installation and execution of Netperf benchmark tool: stop.vl.O.tar 
and linux.vl.O.tar. 

stop.vl.O.tar is used for the installation of netserver process on STOP server, while linux.vl.O.tar 
is used for the installation of netperf client and execution of benchmark tests on the TPE. Both 
are under configuration management within the MYSEA project. 

B.l Installing Netserver 

1. Decompress stop.vl.O.tar 


tar -xf stop.vl.O.tar 


2 . Select mode of installation 

Inetd is used for the setup “M" while daemon is used for the setup “M-". The default 
mode is inetd mode. 


inetd mode: 

./mysea_install_netserver.sh -m inetd 

daemon mode: 

./mysea_install_netserver.sh -m daemon 


3. Optional 

Specify the control port number for netserver to monitor on the network. The default is 
12865. 


./mysea_install_netserver.sh -cp 10000 


Specify the help flag “h" for more options. 


./mysea_install_netserver.sh -h 
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B.2 Installing Netperf 

1. Decompress linux.vl.O.tar 


tar -xf linux.vl.O.tar 


2 . Optional 

Specify the help flag “h" for more options. 

./mysea_install_netperf.sh -h 


B.3 Execute benchmark tests 

1. Installing Netserver and Netperf 

Refer to Section IbH and Section HO for the installation details 

2. Execute benchmark tests 

Provide the necessary flags for the execution of different tests. 


./mysea run netperf.sh 

Flags 

-cp|conPort 

Control port that netserver is monitoring 

-dp|dataPort 

Data port used for transmission of data between Netserver 

and Netperf 

-H Ihost 

IP address or hostname of the targeted machine, where 

netserver server process is residing 

-n |numTests 

Number of times each test is executed 

-t Itest 

Name of the test to be executed 

-tl|testlen 

The length of time in seconds each test is executed. 


3. Collect Results 

Netperf collects the results from the tests and places them in the current directory. The 
default format of the output file is: 

result-[test]-[testlen]-[numTests]-yyyymmddHHMMSS 
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4. Optional 

Specify the help flag “h" for more options. 

./mysea_run_netperf.sh -h 
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APPENDIX C: 
Overview of Results 


This appendix contains the boxplot summaries of all the testing. The boxplots compare the 
results of the benchmark tests executed under all the scenarios. 
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Figure C.5: Boxplots of TCP_RR Tests of all scenarios 
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APPENDIX D: 

Results of IPSec Comparison 


This appendix contains the results of the benchmark tests used for the analysis of the perfor¬ 
mance overhead of IPSec. 
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Figure D.l: Boxplots comparison of IPSec using TCP_STREAM Tests 
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Figure D.2: Histogram comparison of IPSec using TCP_STREAM Tests 
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Figure D.3: Scatterplot comparison of IPSec using TCP_STREAM Tests 


60 































Throughput (lCT3bits/s) 


UDP STREAM 


THROUGHPUT vs Scenarios 



Figure D.4: Boxplots comparison of the performance of IPSec using UDP_STREAM Tests 
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Figure D.5: Histogram comparison of the performance of IPSec using UDP_STREAM Tests 
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Figure D.6: Scatterplot comparison of the performance of IPSec using UDP_STREAM Tests 
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Figure D.7: Boxplots comparison of the performance of IPSec using TCP_CC Tests 
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Figure D.8: Histogram comparison of the performance of IPSec using TCP_CC Tests 


65 

























































TCPCC 

Throughput vs Experiment Number 



Figure D.9: Scatterplot comparison of the performance of IPSec using TCP_CC Tests 
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Figure D.10: Boxplots comparison of the performance of IPSec using TCP_CRR Tests 
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Figure D. 11: Histogram comparison of the performance of IPSec using TCP CRR Tests 
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Figure D.12: Scatterplot comparison of the performance of IPSec using TCP CRR Tests 
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Figure D.13: Boxplots comparison of the performance of IPSec using TCP RR Tests 
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Figure D.14: Histogram comparison of the performance of IPSec using TCP_RR Tests 


71 

























































TCP RR 



Figure D.15: Scatterplot comparison of the performance of IPSec using TCP_RR Tests 
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Figure D.16: Boxplots comparison of IPSec using TCP_STREAM Tests 
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Figure D.17: Histogram comparison of IPSec using TCP STREAM Tests 
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Figure D.18: Scatterplot comparison of IPSec using TCP STREAM Tests 
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Figure D.19: Boxplots comparison of the performance of IPSec using UDP_STREAM Tests 
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Figure D.20: Histogram comparison of the performance of IPSec using UDP STREAM Tests 
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Figure D.21: Scatterplot comparison of the performance of IPSec using UDP STREAM Tests 
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Figure D.22: Boxplots comparison of the performance of IPSec using TCP_CC Tests 
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Figure D.23: Histogram comparison of the performance of IPSec using TCP CC Tests 
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Figure D.24: Scatterplot comparison of the performance of IPSec using TCP CC Tests 
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Figure D.25: Boxplots comparison of the performance of IPSec using TCP_CRR Tests 
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Figure D.26: Histogram comparison of the performance of IPSec using TCP CRR Tests 
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Figure D.27: Scatterplot comparison of the performance of IPSec using TCP CRR Tests 
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Figure D.28: Boxplots comparison of the performance of IPSec using TCP_RR Tests 
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Figure D.29: Histogram comparison of the performance of IPSec using TCP_RR Tests 
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Figure D.30: Scatterplot comparison of the performance of IPSec using TCP_RR Tests 
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APPENDIX E: 

Results of MYSEA Comparison 


This appendix contains the results of the benchmark tests used for the analysis of the perfor¬ 
mance overhead of the MYSEA proxy. 
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Figure E.l: Boxplots comparison of MYSEA processes using TCP STREAM Tests 
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Figure E.2: Histogram comparison of MYSEA processes using TCP_STREAM Tests 
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Figure E.3: Scatterplot comparison of MYSEA processes using TCP_STREAM Tests 
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Figure E.5: Histogram comparison of the performance of IPSec using UDP STREAM Tests 
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Figure E.6: Scatterplot comparison of the performance of IPSec using UDP_STREAM Tests 
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Figure E.8: Histogram comparison of the performance of IPSec using TCP CC Tests 
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Figure E.9: Scatterplot comparison of the performance of IPSec using TCP CC Tests 
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Figure E.10: Boxplots comparison of the performance of IPSec using TCP_CRR Tests 
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Figure E.12: Scatterplot comparison of the performance of IPSec using TCP CRR Tests 
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Figure E.14: Histogram comparison of the performance of IPSec using TCP RR Tests 
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Figure E.15: Scatterplot comparison of the performance of IPSec using TCP_RR Tests 
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Figure E.16: Boxplots comparison of MYSEA processes using TCP_STREAM Tests 
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Figure E.17: Histogram comparison of MYSEA processes using TCP STREAM Tests 
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Figure E.18: Scatterplot comparison of MYSEA processes using TCP_STREAM Tests 
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Figure E.19: Boxplots comparison of the performance of IPSec using UDP_STREAM Tests 
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Figure E.20: Histogram comparison of the performance of IPSec using UDP STREAM Tests 
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Figure E.21: Scatterplot comparison of the performance of IPSec using UDP_STREAM Tests 
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Figure E.22: Boxplots comparison of the performance of IPSec using TCP_CC Tests 
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Figure E.23: Histogram comparison of the performance of IPSec using TCP CC Tests 
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Figure E.24: Scatterplot comparison of the performance of IPSec using TCP CC Tests 
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Figure E.25: Boxplots comparison of the performance of IPSec using TCP_CRR Tests 
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Figure E.26: Histogram comparison of the performance of IPSec using TCP_CRR Tests 
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Figure E.27: Scatterplot comparison of the performance of IPSec using TCP CRR Tests 
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Figure E.28: Boxplots comparison of the performance of IPSec using TCP_RR Tests 
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Figure E.29: Histogram comparison of the performance of IPSec using TCP_RR Tests 


118 



























































TCPRR 

Throughput vs Experiment Number 



Experiment Number 

Figure E.30: Scatterplot comparison of the performance of IPSec using TCP_RR Tests 
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APPENDIX F: 

Results of APS and TPS on different servers 


This appendix contains the results of the benchmark tests used for the analysis of the perfor¬ 
mance overhead of the MYSEA Federation. 
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Figure F.l: Boxplots comparison of APS and TPS on different servers using TCP_STREAM Tests 
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Figure F.2: Histograms of APS and TPS on different servers using TCP_STREAM Tests 
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Figure F.3: Scatterplots of APS and TPS on different servers using TCP_STREAM Tests 
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Figure F.4: Boxplots comparison of APS and TPS on different servers using UDP_STREAM Tests 
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Figure F.5: Histograms of APS and TPS on different servers using UDP_STREAM Tests 
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Figure F.6: Scatterplots comparison of APS and TPS on different servers using UDP STREAM Tests 
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Figure F.7: Boxplots comparison of APS and TPS on different servers using TCP_CC Tests 
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Figure F.8: Histograms of APS and TPS on different servers using TCP CC Tests 
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Figure F.9: Scatterplots comparison of APS and TPS on different servers using TCP CC Tests 
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Figure F.10: Boxplots comparison of APS and TPS on different servers using TCP_CRR Tests 
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Figure F.ll: Histograms of APS and TPS on different servers using TCP_CRR Tests 
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Figure F.12: Scatterplots comparison of APS and TPS on different servers using TCP_CRR Tests 
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Figure F.13: Boxplots comparison of APS and TPS on different servers using TCP_RR Tests 
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Figure F.14: Histograms of APS and TPS on different servers using TCP RR Tests 
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Figure F.15: Scatterplots comparison of APS and TPS on different servers using TCP_RR Tests 


































APPENDIX G: 
UDP Discussion 


The results of UDP_STREAM test show abnormalities in its throughput. Figure [GJ] shows the 
boxplot summary of test results for UDP_STREAM test. The throughput mean values for the 
scenarios when IPSec encryption tunnel is not deployed is smaller than when the IPSec encryp¬ 
tion tunnel is deployed. This is counter intuitive as the presence of IPSec encryption tunnel 
encrypting UDP packets should result in a additional overhead, and thus reduce its throughput. 
Upon analysis of the raw test data for the scenario when NO IPSec encryption is used, we dis¬ 
covered that the throughput is low because of the large disparity between the number of udp 
packets sent by netperf and the number of udp packets received by netserver. Table |G.1| shows 
the number of udp packets sent and received when no IPSec encryption is used. 


In addition, Table |G.2] shows the test results for the scenario when IPSec encryption is used. We 
see a greater than 50% decrease in the number of UDP packets received. 


Table G.l: Examples of UDP_STREAM Test raw data for NO IPSec encryption 


No. 

local send throughput 
(10 3 bits/s) 

remove recv throughput 
(10 3 bits/s) 

local packets sent 

remote packets recv 

1 

960418.23 

998.37 

75035 

78 

2 

960815.57 

25.6 

75066 

2 

3 

960764.19 

153.6 

75062 

12 

4 

960760.61 

25.6 

75062 

2 

5 


6 



The following are the suggested steps for future work. 

1. Use a network analyzer to capture the UDP packets Verify that the number of packets 
captured on both sending and receiving party is accurately reported by netserver. 

2. Analyze the function in source code: recv_omni in nettest_omni.c An error message was 
reported in this function: “ YO! TIMESUP!”. Verify that the system calls that netserver 
use for UDP related tests are valid in STOP OS. 
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Table G.2: Examples of UDP_STREAM Test raw data when IPSec encryption is used 


No. 

local send throughput 
(10? bit s/s) 

remove recv throughput 
(10? bit s/s) 

local packets sent 

remote packets recv 

1 

326865.51 

141674.15 

25538 

11069 

2 

327094.31 

155394.12 

25556 

12141 

3 

329015.93 

149571.31 

25706 

11686 

4 

327903.53 

149546.23 

25619 

11684 

5 


6 
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Figure G.l: Boxplot showing the summary of test results for UDP_STREAM test 
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